Skip to content

feat(pipeline): add verify stage that runs the plan's proof and captures evidence - #18

Open
ci wants to merge 1 commit into
feat/ai-pipeline-skillfrom
feat/pipeline-verify-stage
Open

feat(pipeline): add verify stage that runs the plan's proof and captures evidence#18
ci wants to merge 1 commit into
feat/ai-pipeline-skillfrom
feat/pipeline-verify-stage

Conversation

@ci

@ci ci commented Sep 3, 2026

Copy link
Copy Markdown
Owner

Problem

The pipeline stopped at a clean review. Nothing ran the proof the plan promised (full gate, runtime behaviour, UI or simulator evidence), so the main agent had to do it by hand or trust the implementer's self-report.

Changes

  • New verify stage after review: a fresh engine (--verify, default codex:gpt-5.6-sol:high, env PIPELINE_VERIFY) receives the task, the plan (with its "Exact proof" section), and the implementation report, runs every proof command plus the repo gate, exercises runtime behaviour, and writes evidence files under <run>/evidence/.
  • Structured result verify.json: status (pass|fail|blocked), proof (one entry per numbered proof item from plan.json, validated for coverage, uniqueness, and range), checks (every command run; must be non-empty), evidence (files that must exist inside <run>/evidence/), notes, plus the tree id and proof snapshot it certified. Anything but a full pass halts with exit 3.
  • Verify only certifies the exact tree the last clean review saw: the review stage records reviewed_tree; verify refuses a different tree at start and halts if the committed tree changes during the stage (a verifier that "fixes" and commits cannot self-certify). run sends a run whose tree moved back to review, halts on a dirty workspace, and re-verifies when task/plan inputs changed since the verified run. Re-entering review or verify rewinds the completion pointer only after the workspace is known clean, so a halted rerun is retried by run.
  • Plan drift: plan.json proof is canonical; a plan.md edit without a proof update halts verify until the list is changed or verify --accept-proof confirms it. Baselines move to the last accepted or verified pair. A fresh implement after review/verify resets its commit baseline.
  • Ordering: verify requires a completed review; run now goes through verify by default (--until accepts it). Runs created before this change fall back to the default verify engine.
  • summary.md gains a Verification section listing checks and evidence paths; SKILL.md documents the stage, halts, permissions, and how to route a failed verification (fix and rerun verify, or amend task.md and rerun implement + review).

Tests

  • Real run on the toy repo (codex engine): four checks pass (uvx pytest -q, three CLI behaviours), three evidence files written, status verified, summary section rendered.
  • Unit checks via import: halt on fail, on blocked, on pass with a failed check; completion on a clean pass; ordering guard when review is not complete; STAGES order.
  • Two real verify runs on the toy repo with the final contract (numbered proof, evidence validation): all proof items and checks pass, evidence files present, status verified.
  • Unit checks via import: halts on fail/blocked/failed check/missing or duplicate proof index/empty checks/invalid evidence/plan drift/reviewed-tree mismatch/tree change during verify; --accept-proof; baseline refresh; run rewind on tree change, halt on dirt, re-verify on input change; review re-entry rewind; summary staleness for tree, proof, and input changes.
  • autoreview --reviewers codex:gpt-5.6-sol:xhigh,grok:grok-4.6:xhigh: five rounds (the cap). Rounds 1-4 findings all fixed. Round 5 raised three P2/P1 items (dirty workspace after verify treated as done by run; summary not stale on task/plan prose edits; implement baseline retained across a post-verify rerun); all three fixed after the cap without a further review round. Grok reported clean in rounds 2-5.

@ci

ci commented Sep 3, 2026

Copy link
Copy Markdown
Owner Author

This PR is part of a stack:

  1. feat/autoreview-grok-engine
  2. feat/ai-pipeline-skill
  3. feat/pipeline-verify-stage <-- this PR

Created with jjpr

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant